A semiconductor memory device having a three-dimensional structure is proposed, wherein memory holes are formed in a stacked body in which multiple electrode layers are stacked, and a charge storage film and a semiconductor film are provided in the memory hole to extend in a stacking direction of the stacked body. The semiconductor memory device includes multiple memory cells connected in series between a drain-side select transistor and a source-side select transistor. The electrode layers are gate electrodes of the drain-side select transistor, the source-side select transistor and the memory cells, and are formed by removing a portion of the stacked body via a slit of the stacked body and then filling a metal material in the removed portion. When removing the portion of the stacked body, there is a possibility that variation arises in the thickness of film formed in each memory hole depending on a distance between the memory hole and the slit. Thus, it is concerned that failure occurs in the operation of the memory cell.